Reliable and Locality Driven Scheduling in Hadoop

The increasing use of computing resources in our daily lives leads to data being generated at an unprecedent rate. The computing industry is being repeatedly questioned for its ability to accommodate the unpredictable growth rate of data, and its ability to process them. This has encouraged the development of cluster based data-intensive applications. Hadoop is a popular open source framework known for its massive cluster based data processing power. Hadoop is widely used in the computer industry because of its scalability, reliability, ease of use, and low cost of implementation. Cloud computing in the recent years has gained increasingly popularity by its cost-efficient and flexible way to leverage the power of commodity hardware. Hadoop based services on the Cloud have also emerged as one of the prominent choices for smaller businesses. However, evidence in the literature shows that faults on the Cloud do occur and normally result with performance problems. Hadoop hides the complexity of discovery and handling failures from the schedulers, but the expenses of failure recovery rest entirely on users, regardless of root causes. We systematically assess these expenses through a set of experiments, and argue that more effort to reduce this cost to users is desirable. We also analyze the drawback of current Hadoop’s mechanism in prioritizing failed tasks. By trying to launch failed tasks as soon as possible regardless of locality, it significantly increases the execution time of jobs with failed tasks, due to two reasons: 1) available slots might not be free up as quickly as expected and 2) the slots might belong to machines with no data on it, introducing extra cost for data transferring through network, which is normally the most scarce resource in nowadays’ data centers. This thesis then introduces a new algorithmic approach called the waste-free preemption. The waste-free preemption saves Hadoop scheduler from choosing solely between kill, which instantly releases the slots but is wasteful, and wait, which does not waste any previous effort but suffers from the two above mentioned reasons. With this new option, a preemptive version of Hadoop default scheduler (FIFO) is implemented. The evaluation demonstrates the effectiveness of the new feature by comparing its performance with the traditional Hadoop mechanism.